Multi-Task and Multi-Modal Learning for RGB Dynamic Gesture Recognition

Gesture recognition is getting more and popular due to various application possibilities in human-machine interaction. Existing multi-modal gesture systems take data as input improve accuracy, but such methods require modality sensors, which will greatly limit their scenarios. Therefore we propose an end-to-end multi-task learning framework training 2D convolutional neural networks. The can use the depth accuracy during save costs by using only RGB inference. Our trained learn a representation for learning: segmentation recognition. Depth contains prior information location of gesture. it be used supervision segmentation. A plug-and-play module named Multi-Scale-Decoder designed realize segmentation, two sub-decoder. It lower stage higher respectively, help network pay attention key target areas, ignore irrelevant information, extract discriminant features. Additionally, MSD are performance. Only without required Experimental results on three public datasets show that our proposed method provides superior performance compared with existing frameworks. Moreover, other CNN-based frameworks also get excellent improvement.